Computational Vision
U. Minn. Psy 5036
Daniel Kersten
Lecture 15: Geometry & surfaces: depth & shape

Initialize

Off[General::spell1];

SetOptions[ArrayPlot,ColorFunction->"GrayTones",DataReversed->True,Frame->False,AspectRatio->Automatic,Mesh->False,PixelConstrained->{1,1},ImageSize->Small];

SetOptions[DensityPlot,ColorFunction->"GrayTones",Frame->False,AspectRatio->Automatic,Mesh->False,ImageSize->Small];

Outline

Last time

Extrastriate cortex--overview

Scenes from images, scene-based modeling of images

Today

Geometry, shape and depth: Representation & generative models

Lambertian model

Surfaces, geometry & depth

Introduction

Recall two major tasks of vision: "Knowing what is where just by looking"--Marr's definition of vision. 
We introduce the problem of extracting geometrical information about the world.

General issues: Between-object, viewer-object, within-object geometry.
Coarse vs. dense estimations of geometrical relations.

The projection constraint

The fundamental geometrical constraint in vision is that 3D points in the world are projected onto 2D surfaces--one on each retina. 
The figure below illustrates the geometry. We use the convention of representing the image in the conjugate plane to the retina. The origin represents the nodal point of the eye--where light rays pass through undeviated. (We are making the simplifcation of assuming a pin hole model of the eye where we don’t have to worry about optical aberrations). The 3D coordinates of a point in the world is {X,Y,Z} which project to {x,y} in the conjugate image plane.

If we let the focal length f=1, then:

which corresponds to our intuition that points that are farther away would get mapped to points closer to the origin. Later in the course, we will describe how to model projections using “homogeneous coordinates” as well as other geometric transformations including rotation, translation, and scale. We will also look at the kinds of constraints that result, for example that straight lines should map to straight lines.

What happens if the focal length is very long, i.e. on the scale of the distance Z?

Two basic classes of geometrical information for vision

Scene geometry--Spatial layout, large-scale surface structure

Where are objects relative to the viewer? 
		Where are they relative to each other? Relative to a frame? (e..g. ground plane)

Object geometry--Surfaces & shape, small scale surface structure

How can we describe objects themselves in terms of their intrinsic geometry--shape?
		What is the relationship of features/parts of objects to each other? 
		Extrinsic vs. intrinsic geometrical descriptions

Scene geometry: Spatial layout

Absolute depth: Viewer-object relations

Distance of objects or scene feature points from the observer.

"Physiological cues": Binocular convergence--information about the distance between the eyes and the angle converged by the eyes. Crude, but constraining.  Errors might be expected to be proportional to reciprocal distance. 
	diopters (or  meter-angles) = 1/(meters)
Closely related to accommodative requirements.
	
"Pictorial cues"--familiar size

Pattern of errors can depend on how human absolute depth is assessed (e.g. verbal estimates vs. walking) (Loomis et al., 1992)

Important for navigation and reaching. (Marrotta and Goodale, 2001).

Relative depth: Object-object relations

Distance between objects or object feature points. Important for scene layout, planning actions, navigation planning.

Processes include: Stereopsis (binocular parallax) and motion parallax (e.g. http://psych.hanover.edu/krantz/motionparallax/motionparallax.html).

Also information having to do with the "pictorial cues": 
	occlusion (interposition), 
	transparency, 
	perspective, 
	proximity luminance, 
	focus blur, 
	familiar size & "assumed common physical size", 
	"height in picture plane", 
	cast shadows, 
	texture & texture gradients for large-scale depth & depth gradients.

The pictorial cues have been a big challenge for computational vision, because accuracy is closely tied to segmentation and the image parsing problem

Examples of pictorial information for depth

There are lots of illusions that illustrate the interactions between size and depth, e.g. the Ponzo illusion.

For some depth from shadow illusions, see: http://gandalf.psych.umn.edu/~kersten/kersten-lab/demos/shadows.html  and for interactions between motion parallax and transparency/occlusion see: http://gandalf.psych.umn.edu/users/kersten/kersten-lab/demos/transparency.html

The representation problem: How should distance be represented?

Absolute units, relative units, or ordinal? Answers may depend on:

	 1) type of cue--interposition vs. disparity (cf. Engel et al., 2006)
	 2) task (e.g. reaching, planning, navigation, instantaneous heading, recognition...)
	 3) requirements for cue integration--integration requires common representation

More later

One can list over a dozen cues to depth. Later, we'll study theories of integration (e.g. stereo + cast shadows). Also theories of cooperativity (e.g. motion parallax <=> transparency). One of the problems is to understand how cues that seem to be naturally related to one type of representation can be integrated with those from another.

Object geometry: Shape

Important for determining what an object is. Although object recognition can make use information of such as material or context, shape is particularly informative.

We also need to distinguish between shape in the projected image and its relationship to object shape in the world. A static shape on the retina is 2D, objects in the world can be roughly confined to 2D (e.g. a piece of paper) or be 3D.

We'll put off the problem of precisely defining shape for the moment. It isn't that it can't be done, but we'll see that, like depth, its definition is closely tied to what image information is available, what the shape information will be used for, and what needs to be discounted to get there. So let's look at some examples of image information for shape.

Contours & region information

1D "lines" vs. 2D "fields or regions". I.e. object contours vs. region information. Here are some examples.

Contours and region processes in the 2D image support the inference of both 2D and 3D shape. Let’s look more closely at these.

Cues to shape: Contour based

Contour based--contours at object boundaries (depth discontinuity) and at sharp object bends (orientation discontinuities, smooth self-occluding contours). 

The famous Kanizsa triangle below illustrates contours at apparent depth discontinuities. The illusion illustrates the effectiveness of contours in conveying 2D shape, and occlusion relationships.

Cartoons illustrate the effectiveness of line-drawings.

Visual inferences about 3D shape can be made with surprisingly little image information. 

The contours below are interpreted as self-occluding contours, where the surface smoothly wraps out of view. Is the "worm" flat or round? Is the vertical rod flat or round?

http://jov.arvojournals.org/article.aspx?articleid=2121569

Cues to 3D shape: Region based

Region based: shading, texture gradients, disparity gradients & fields, motion parallax fields

Example of shading & shape

Example of texture and shape

...but the reasons why we see shape given a texture aren't always obvious:

Are the square elements all the same size? http://www.ritsumei.ac.jp/~akitaoka/Bulge02.jpg

Later, we'll look at the interactions between contour and region information in the perception of shape.

Local vs. global shape

If you pick a point on one of the above "croissants", it can be characterized in terms of its local surface orientation (a viewer-centered, extrinsic representation, which we discuss below) or its local curvature (elliptic, hyperbolic, cylindrical, which are measures intrinsic to the surface, independent of viewer coordinate system). 

But you can also characterize the whole shape, i.e. "croissant-like" or "banana-like". 

One can ask whether the visual system is sensitive to global properties or statistical summaries such as aspect ratio, compactness. Shape can also be described qualitatively, and complex objects built out of relational descriptions (e.g. Biederman, 1985). We go more into depth below.

Features and the shape representation itself can be sparse (coarse) or dense. Given a coarse set of  features (the small ellipses below), the human visual system interpolates shape between features, often with “phenomenal” smoothness -- i.e. your perceptual experience reflects an underlying process of surface interpolation through the features. This is problem is perceptually important and of theoretical interest. How to go from a set of sparse measurements in a region to a dense field? We’ll see more of that later.

The shading on the above croissants provides continuous, dense cues.

Local shape fields & dense estimation

From dense depth to shape, normals & curvature

Below we will start with a mathematically simple representation of depth from the viewer: z = f(x,y), and show how to derive a simple dense shape measure in terms of the rate of change of depth from the viewer. This a viewer-centered depth representation. Later, we will discuss intrinsic object shape measures such as curvature.

For an example of the interaction between shape and viewpoint see: http://www.michaelbach.de/ot/sze_shepardTables/index.html

Local dense representations of surface regions

Shape-based generative modeling of images

We'll start with a discussion of dense local representation of shape in terms of surface normals, and understand how shape changes influence image intensity changes. And from there, the inverse problem of seeing shape from shading in the next lecture.

There are two aspects to image formation. The first has to do with geometrical manipulations that describe how surface points project to image points (which we will study later via matrix transformations on homogeneous coordinates). The second aspect is the photometric determination of intensity at each image point from a scene description. In order to understand shape from shading, we need to understand the constraints implicit in the forward optics problem of  "shading from shape"--i.e. the generative model. Here we can benefit from research in attempts in computer graphics to obtain photorealism.

Image formation constraints can be obtained by understanding how material properties, shape, and illumination interact to form an image. This is part of the field of computer graphics.  The quest for physical realism in reasonable computing time is still a challenge in computer graphics (Greenberg, 1999; for eary history, see: Blinn, J. F., 1977; Cook, R., & Torrance, K.,  1982).

One of the earliest models is the Lambertian shading equation which describes how intensity is distributed for curved matte surfaces with constant reflectance (e.g. arbitrarily defined as 1):

L(x,y) = (Overscript[E, ⋀])·(Overscript[N, ⋀])(x,y)

E(x,y) is a vector representing light source direction. If at infinity, it has just two degrees of freedom and is constant over the surface. If light source intensity is normalized, then the E vector is a unit vector. The above expression assumes a light source at infinity. Alternatively its vector length can be used to  indicate the strength of the illumination.

Surfaces can have varying reflectance, r(x,y) representing that fraction of light reflected (varying from 0 for a black pigment to 1 for a white pigment), and then the shading equation becomes:

L(x,y) =  r(x,y) (Overscript[E, ⋀])·(Overscript[N, ⋀])(x,y)

An example of a simple computer graphics lighting model--"matte" & "plastic" world

An early elaboration of the lambertian model for shading, based in part on physics, and in part on heuristics and "beauty pageant" observations, describes image luminance in terms of ambient,  lambertian, and specular components:

L =  (Subscript[r, a])(Subscript[E, a])+ r (Overscript[(Subscript[E, p]), ⋀])·(Overscript[N, ⋀]) + (Subscript[r, s])((((Overscript[R, ⋀])·(Overscript[L, ⋀])))^n)

where the lower case r's are the reflectivities (0 means no reflectance of the illumination, and 1 means complete reflectance), and the unit vectors   E, N, and  L, point in the directions of the light source, surface normal, and viewpoint respectively. Ep and Ea, are  the strengths of the point source, and ambient illumination, respectively.   R points in the direction that a ray would go if the surface was a perfect flat mirror, i.e. purely specular. (Overscript[E, ⋀])·(Overscript[N, ⋀]) ∝ cos((Subscript[θ, i])) and  (Overscript[R, ⋀])·(Overscript[N, ⋀]) ∝ cos((Subscript[θ, r])), where (Subscript[θ, i]) and (Subscript[θ, r]) are the angles of incidence and reflection, respectively. For a flat mirror surface, (Subscript[θ, i]) = (Subscript[θ, r]).

The form for the specular term, ((((Overscript[R, ⋀])·(Overscript[L, ⋀])))^n) ,  is due to Phong, who pointed out that  n could be used to control the degree of specularity. High values of n correspond to a perfect mirror surface, values between 1 and 200 are typically used to add some gloss to the rendering. Surfaces then look more plastic, or metallic.

Perception-based fudge factors

One can try to build in “effects” that enhance realism without explicit consideration of the physics. For example,

where the R+K term is a "fudge" term, reflecting common experience (not physics) in which surfaces (of the same reflectance, and orientation) that are farther away from the viewer are dimmer. K is a constant, and R is the distance from the object to the viewpoint. This is a weak, but nevertheless useful constraint for vision. A closely related idea is "proximity luminance" cues in vision (see above, and also Dosher, B. A., Sperling, G., & Wurst, S. A., 1986). The assumption is that one’s viewpoint is likely to be close to the light source direction than opposite it.

Limitations

For more on rendering, including a physically correct phong-like model, see: Larson, G. W., & Shakespeare, R. (1998). (We’ll look more closely at physically correct photometric models later when we consider problems of material perception.)

Note that as is, there are severe limitations to the above image formation models. One of the major problems is  the absence of cast shadows. 

Physically based models become much more complicated when one has to take into account multiple bounces of light off of surfaces, for example through ray-tracing. (See Computer Graphics: Principles and Practice (3rd Edition); Fundamentals of Computer Graphics by Peter Shirley et al. and Visual Perception from a Computer Graphics Perspective by William Thompson, Roland Fleming, Sarah Creem-Regehr and Jeanine Kelly Stefanucci (Jun 2, 2011)). 

While the above equation also not take into account how light bounces between objects (mutual illumination or indirect lighting)--the ambient “fudge” term is the crude approximation to model the overall effect of mutual illumination.

Material properties can be much more complicated than Lambertian plus a Phong specular term. Illumination patterns can also be much more complicated, as illustrated everytime you look at a shiny object. More on the complexities of the image formation model later.

Preview of the shape-from-shading problem

Later we will study the "shape-from-shading" problem. If one represents shape in terms of a dense distribution of surface normals, then a simplified version of the formal problem is to estimate N(x,y) given data L(x,y) such that the following simplification of the above equations holds:

As it stands, this set of equations (one for each location x,y) is underconstrained or "ill-posed". Even if we knew the light source direction E, for every image intensity L, there are two numbers to estimate for N. Assuming the light source is a point at infinity simplifies things (same two numbers E for all surface points). Assuming surface smoothness and integrability also constrains the solution. But more on this later.

Representing shape: global representations

We pointed out above that a central issue in object perception is how the shape of an object is represented by the visual system. Shape may be represented in a variety of ways that depend on the visual task and the stages of processing in a given task. 

Questions about shape representation can be classified along several dimensions. Two central questions have to do with whether the representation is local or global, and whether the representation depends on viewpoint.

Global and local representations of shape.  A global representation of solid shape consists of a set of parameters or templates that describe a class of surfaces or ``parts". Several theories of object recognition assume that objects can be decomposed into elementary parts. These parts are drawn from a limited set of elementary shapes, such as generalized cylinders (Marr and Nishihara, 1978), geons (Biederman, 1987) or superquadrics (Pentland, 1990).  Shapes can be characterized by their skeletons (e.g. Wilder, J., Feldman, J., & Singh, M., 2011). 

Some global representations have the property that a change of one parameter which describes a part will affect the whole shape.  An example of this are spline representations.

Manipulate[Graphics3D[Tube[BezierCurve[{{0,0,0},{1,1,1},{2,t,s},{3,0,2}}],0.2],SphericalRegion->True,ImageSize->Small],{{s,1.0},.1,2.0},{{t,-1.0},-2,2.0}]

In comparison, a local representation is a dense characterization of shape, such that a change of a parameter at one spatial location will not affect the shape at another location. A surface normal vector at each surface location is an example of a dense representation.

Viewpoint dependency in shape representations. A second major question is whether the shape representation depends on viewpoint. This debate has arisen in the context of models of object recognition  (Tarr & Bülthoff, 1995). Local viewpoint dependent descriptors such as surface normals (or slant and tilt, see below) have an early history (Todd & Mingolla, 1983; Mingolla & Todd, 1986; see below). But view-dependent descriptors have disadvantages for some tasks. One problem with slant and tilt is that the local slant of an oriented plane varies with viewpoint. So we have a discrepancy between apparent global flatness and the local variation in slant (Mamassian, 1995).  

For part-based object recognition, it would seem best to do part extraction using local view-independent descriptors. It is also reasonable that view-independent shape descriptors could support other types of visual processes. For instance, the manual prehension of an object requires one to locate stable grasp points on the surface, a task which only makes sense in an object-centered frame of reference.  Nevertheless, the visual information is of course firstly described in a viewer-centered frame of reference, and the fundamental issue then becomes the tranformation of viewpoint dependent into viewpoint independent representation (Andersen, 1987). One local, intrinsic representation of solid shape describes the second order depth variation of the surface (Besl and Jain, 1986), or equivalently, the first order orientation variation (Rogers and Cagenello, 1989).  For more on shape representations, see Koenderink (1990).

Compositions. When we discuss the problem of object recognition later in the course, we’ll look at ways of describing objects in terms of “compositions”, in which one  builds a global representation from “resuable” features and parts and their relationships. These are sometimes called “structural descriptions”. Such descriptions have a number of advantages including the ability to change a part description without affecting the whole, and when incorporated into a hierarchical model, the ability to represent a rich set of a semantically meaningful variations.

Gradient space & surface normals

Let’s look more closely at local, dense, viewpoint dependent representations. We begin with  surface normal representation that we saw above: N(x,y). 

Imagine a small planar surface patch. One way of representing shape locally and with respect to the viewpoint of the observer, is to use gradient space:

Let ϕ(x,y,z)=f(x,y)-z  Then,

is a normal vector to the surface at (x,y,f(x,y))  :

Gradient space is defined by the mapping of (p,q,-1)-->(p,q), i.e. the orthographic projection. If we normalize to unit length, we get an expression for the surface normal at location (x,y):

N(x,y) = ((p/(Sqrt[(p^2)+(q^2)+1])),(q/(Sqrt[(p^2)+(q^2)+1]))).

Slant and tilt

Shape can also be described by slant and tilt values across the coordinates of an image.  Again, imagine a small planar surface patch being viewed along the z-axis. 

(Note: Sign convention can differ from one study to the next, e.g. whether the viewer is looking from a point on +z or -z.)

Slant and tilt measures are used in perceptual studies of both local dense estimation, and global near-planar surface attributes, e.g. for large-scale layout. 

Let N be the (un-normalized) surface normal expressed in terms of p and q:

Here are the important intuitive interpretations.

The slant is the inclination of the surface relative to the viewer and is related to p and q by:

A slant of zero means that the surface is "fronto-parallel".

The tilt is the direction of steepest descent away from the viewer:

Note: Sqrt[q^2+p^2] is the rate of change of z in the direction of maximum change (steepest descent).

How to choose a representation?

How do we know what is the best representation to use? Slant and tilt seem to be important from a perceptual point of view. 
But one reason gradient space is useful is that it is related to relative depth in a straight forward way. 

We can get back to distance by integrating:

gives the relative distance, and a constant is lost in the process.

For examples of perceptual studies of local shape representations see: Koenderink, 1990 (and the many papers/book by him and his group);  and also Mamassian & Kersten 1996, Mamassian, Knill & Kersten (1996).

Using Mathematica to go from depth to normals to image intensities using the Lambertian model

This section gives you some practice using the Lambertian scene-based generative model.

Range data defines surface list

Range data to define rface list -- BIG 64x64 file. 
(Originally from: http://sampl.eng.ohio-state.edu/)

rface = {{2,2,2,2,2,2,2,2,2,1,3,7,54,89,99,110,121,128,136,143,149,156,164,171,177,183,187,191,194,194,197,198,198,197,196,194,192,188,184,178,172,165,157,152,145,139,132,126,118,112,103,94,81,62,7,2,2,2,2,2,3,2,4,9},{0,0,0,0,0,0,0,0,0,0,0,5,57,98,109,122,133,141,150,158,165,172,180,188,194,200,205,208,211,213,215,216,216,216,215,214,210,206,202,198,192,183,175,168,160,152,144,138,128,120,111,101,86,45,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,0,0,7,67,108,118,128,141,149,157,165,172,179,186,194,200,205,211,214,217,220,220,219,220,220,220,219,216,212,209,205,199,191,183,174,166,158,150,141,132,125,114,103,90,45,1,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,1,0,1,3,49,97,120,140,150,158,165,172,180,187,194,201,208,213,220,222,224,226,226,226,226,226,226,225,224,222,218,214,208,201,192,182,174,166,157,148,139,131,120,109,95,42,2,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,1,1,0,2,60,139,150,158,165,172,179,187,194,200,207,215,220,226,229,231,232,232,231,229,230,230,230,230,229,225,222,217,210,200,191,182,174,164,157,147,137,127,116,103,24,0,0,0,0,0,1,1,0,0,0},{0,0,0,0,0,0,0,0,1,0,0,0,5,86,159,157,165,172,178,185,192,199,206,213,220,225,230,233,235,235,235,234,233,233,234,235,235,233,231,228,223,217,207,198,189,181,171,165,155,145,134,122,110,17,1,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,0,0,0,59,148,157,165,172,178,184,190,196,203,210,218,223,228,233,236,236,236,237,237,236,236,237,238,238,237,235,232,227,223,215,204,195,188,179,171,163,154,144,131,118,20,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,1,0,0,48,141,160,162,169,176,182,188,194,200,207,214,220,226,228,234,236,238,240,239,238,237,238,239,240,239,238,237,234,230,225,219,209,200,192,185,177,171,162,151,140,127,22,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,0,16,117,159,160,167,175,180,186,192,198,204,209,215,220,226,231,234,236,239,241,240,239,237,236,239,240,241,240,238,235,230,226,219,212,204,196,189,182,177,168,160,150,139,24,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,2,83,149,155,164,172,177,183,189,193,199,205,210,215,219,226,230,234,236,239,240,240,240,238,237,240,242,241,240,238,234,230,226,220,213,206,199,192,186,179,173,166,157,147,56,1,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,21,141,153,160,167,173,180,186,191,196,200,205,210,215,220,224,229,233,236,238,240,240,239,237,236,238,241,241,240,237,233,229,225,220,213,207,201,195,189,183,176,169,162,153,117,11,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,0,87,146,155,164,170,178,183,188,193,197,201,206,210,215,219,223,226,230,234,237,238,238,238,235,237,238,239,239,236,232,230,228,223,218,213,207,202,197,192,185,178,173,166,156,148,57,1,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,14,132,151,159,165,170,178,184,189,193,198,201,206,210,214,218,223,225,227,229,232,234,235,236,236,237,236,235,233,230,228,227,224,219,215,211,207,202,198,193,188,182,175,169,160,151,114,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,43,135,154,161,167,173,179,185,189,194,199,202,205,208,213,217,221,224,226,225,227,230,233,233,235,233,231,229,228,227,226,224,221,217,214,211,208,203,199,195,190,185,177,171,162,156,138,28,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,94,144,154,163,169,175,181,187,191,195,198,201,205,208,212,215,219,222,225,226,228,229,230,232,231,232,231,230,228,227,225,223,220,216,213,210,207,202,199,193,190,185,178,171,165,158,145,81,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,120,150,156,165,171,175,180,187,192,195,198,200,203,205,209,213,218,221,225,228,230,230,231,231,231,233,231,231,230,228,224,221,218,214,212,208,206,202,200,195,191,186,179,173,166,160,153,112,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,17,121,152,159,165,171,175,180,184,191,195,198,200,201,204,207,211,216,222,226,229,231,231,232,232,232,232,231,231,230,228,224,221,216,212,208,207,204,202,199,196,192,186,179,174,167,162,154,127,17,0,0,0,0,0,0,0},{0,0,0,0,0,0,1,45,128,151,160,167,173,177,182,187,191,195,197,198,199,201,202,207,214,220,225,228,229,230,232,233,231,232,232,231,229,227,223,220,215,209,206,204,202,200,198,197,190,185,179,174,167,162,155,133,46,0,1,0,1,0,0,0},{0,0,0,0,7,28,47,94,134,150,160,167,171,176,181,186,190,195,197,198,199,198,198,202,212,218,223,226,229,231,232,232,232,231,230,230,228,226,222,218,212,205,202,202,201,200,199,197,191,185,179,174,170,164,155,137,55,0,0,0,0,0,0,0},{0,1,2,19,43,58,63,99,135,150,161,168,172,177,182,186,189,195,197,198,200,200,200,204,211,218,223,226,229,231,232,232,233,232,230,229,228,227,222,217,209,202,201,200,201,200,199,195,189,184,180,175,170,163,155,143,97,10,6,0,0,0,1,0},{0,1,21,43,54,63,70,102,133,150,161,168,174,178,183,186,189,194,196,198,201,203,205,209,212,217,223,227,230,232,234,233,233,234,234,232,229,227,221,215,209,204,204,202,201,200,199,195,190,182,177,174,170,164,154,141,106,64,50,21,2,0,0,0},{1,14,35,46,48,55,68,105,133,151,161,168,175,179,184,187,190,193,195,198,201,202,205,210,216,219,222,226,228,229,230,230,229,230,230,230,227,224,219,214,211,207,205,203,202,200,198,193,188,182,179,175,170,164,154,139,107,67,65,58,30,5,1,0},{4,22,29,38,47,49,56,102,132,149,161,169,176,180,184,188,189,191,193,196,199,202,206,211,217,222,226,227,228,229,230,232,230,229,228,227,226,223,219,216,213,209,205,203,202,200,197,192,187,183,179,176,170,163,153,137,107,75,71,61,50,34,1,0},{12,21,18,20,27,37,49,99,130,146,162,171,177,180,184,188,190,191,192,193,196,200,206,211,216,221,227,230,231,232,232,231,232,232,231,230,228,226,222,218,213,210,204,202,199,198,196,192,187,184,179,176,171,164,153,134,107,73,60,53,53,50,21,1},{15,14,14,13,13,16,26,95,131,146,161,169,178,182,186,190,192,193,193,193,193,195,200,208,214,219,223,227,229,230,228,226,228,230,230,231,228,225,221,217,212,206,202,199,198,197,195,190,187,184,180,177,171,164,151,132,104,61,51,56,53,47,35,7},{13,11,13,16,15,10,22,94,129,143,160,170,177,182,188,190,192,193,194,194,193,192,194,200,209,215,220,224,226,226,225,225,225,227,227,227,225,222,217,212,205,199,197,196,196,195,194,191,188,185,181,178,172,164,149,128,101,59,54,46,37,35,32,15},{12,10,14,15,14,12,30,96,126,143,159,168,177,182,188,191,193,194,195,196,196,194,192,195,201,209,217,221,225,225,225,224,224,225,224,224,221,217,212,205,197,195,195,196,197,195,194,192,190,187,182,179,171,162,147,127,91,32,29,26,27,30,28,20},{11,10,14,15,16,16,36,97,126,144,157,166,177,182,186,192,193,194,196,196,197,196,194,194,199,208,214,217,223,224,227,228,226,225,224,221,216,211,204,198,196,195,195,196,197,196,195,192,189,187,182,178,170,160,143,124,86,21,21,24,28,26,24,21},{9,6,13,17,17,24,50,99,126,142,155,165,176,182,186,190,194,195,197,197,197,197,197,196,202,215,216,215,222,228,238,242,235,226,221,214,212,212,203,197,196,197,197,197,197,195,195,193,190,186,183,177,169,160,140,121,84,21,19,24,26,23,20,18},{6,2,10,16,21,35,63,103,126,140,153,163,172,180,186,190,193,194,196,197,197,197,197,196,207,225,226,225,231,236,243,243,241,233,224,220,220,219,206,197,198,198,198,198,198,197,195,192,190,187,183,174,165,156,137,119,76,22,22,24,25,20,18,20},{5,1,4,12,24,22,45,102,127,140,151,160,169,176,183,189,193,195,196,197,197,196,197,195,209,226,233,236,241,243,244,244,244,242,233,232,229,223,207,198,197,198,198,198,197,195,194,192,189,186,181,172,162,151,134,117,70,25,24,24,26,18,15,16},{6,1,3,10,18,2,29,103,129,140,149,157,165,172,179,185,190,194,195,196,197,196,198,197,202,224,236,240,243,248,248,249,249,246,239,235,231,225,202,196,198,198,197,198,197,195,194,191,187,184,176,168,159,147,132,115,80,36,29,26,26,12,10,13},{6,4,4,12,18,1,31,105,130,138,148,156,162,168,175,181,186,189,193,195,196,197,198,198,199,209,233,244,249,251,251,252,252,249,244,240,233,213,199,199,199,199,198,197,195,193,192,189,185,179,171,164,156,146,131,113,60,28,33,29,20,5,7,12},{18,14,14,23,25,0,44,110,128,136,146,153,160,165,171,176,181,186,188,189,192,194,195,197,199,204,221,241,250,252,254,254,254,252,246,240,216,201,199,200,199,198,194,194,193,190,188,184,180,174,167,161,154,144,130,109,36,7,33,24,10,3,9,13},{21,28,29,39,29,0,69,120,127,135,143,150,157,162,166,170,174,177,180,182,185,188,191,192,196,202,213,233,245,251,254,255,254,251,244,226,206,201,199,196,194,194,192,191,189,186,184,179,175,170,164,159,151,141,129,110,40,1,17,17,4,4,12,11},{19,26,31,20,6,0,73,121,128,135,142,147,152,157,162,167,171,173,174,175,175,179,184,187,190,195,205,225,239,246,249,250,248,244,231,213,204,198,195,191,188,185,184,181,181,181,178,176,170,167,162,158,149,138,129,110,40,0,9,16,8,11,15,9},{2,10,15,5,0,0,73,122,128,136,142,146,149,154,160,167,174,179,179,180,179,178,178,179,182,181,189,210,231,239,242,242,239,232,217,203,195,189,184,179,177,175,175,174,174,175,173,173,169,166,160,156,147,137,129,110,39,1,11,25,20,18,18,3},{0,0,0,0,1,0,75,122,128,137,144,148,151,156,161,167,171,176,179,183,184,184,183,181,178,175,176,191,220,232,236,234,232,223,203,184,178,178,177,179,181,182,182,180,179,176,172,168,164,162,157,152,145,137,128,109,40,0,8,30,27,20,12,1},{0,1,0,0,0,0,70,121,129,138,146,151,156,160,164,168,170,172,173,176,178,181,182,180,178,176,173,181,212,226,229,229,226,211,183,172,177,178,183,185,186,187,184,181,179,176,171,164,159,157,154,150,144,136,127,108,38,0,15,34,27,16,1,0},{1,0,0,0,1,0,46,112,130,138,147,153,159,162,166,171,175,178,179,180,180,179,177,176,177,175,174,176,202,220,224,224,221,199,173,172,175,179,180,181,179,179,178,175,173,171,168,165,162,158,154,150,144,135,124,105,41,0,6,15,11,3,0,0},{1,0,0,0,0,0,34,107,130,136,147,156,162,168,171,174,178,181,180,182,183,185,183,181,179,176,178,181,195,210,216,217,210,188,174,173,175,177,177,179,181,182,180,178,175,173,170,167,163,160,156,152,142,132,122,99,21,0,0,1,0,0,0,0},{0,0,0,0,0,1,31,105,130,136,147,158,166,173,176,179,183,184,184,185,185,185,185,186,187,186,192,192,192,200,206,205,197,187,182,181,180,180,181,183,184,185,183,181,178,176,173,169,165,163,158,152,138,129,121,79,0,0,1,0,0,0,0,0},{0,0,0,0,0,0,56,114,128,137,146,156,167,176,181,184,189,190,192,193,193,192,192,195,198,202,200,196,195,200,202,200,197,192,192,192,189,186,184,184,184,184,184,183,181,180,178,173,169,165,158,148,136,128,113,55,0,1,0,0,1,0,0,1},{0,1,0,1,0,5,74,117,126,136,144,154,165,174,181,186,190,193,195,196,196,197,196,197,199,200,198,198,199,202,202,203,199,196,198,202,202,197,193,190,189,189,188,187,185,183,181,176,171,168,156,147,135,126,104,13,0,0,0,1,1,0,0,0},{0,0,0,0,0,18,83,112,124,135,144,153,162,168,175,183,186,189,189,189,192,195,194,194,195,197,198,200,200,204,204,203,202,200,198,197,197,197,198,198,197,197,194,192,189,187,184,179,172,165,155,144,133,124,96,2,0,0,1,0,0,0,0,1},{0,0,0,0,0,26,86,112,123,135,145,153,161,167,174,179,183,186,187,187,190,192,192,194,197,197,198,200,202,203,201,203,202,201,199,198,196,196,196,196,198,197,195,192,191,188,183,177,171,162,150,142,130,119,95,6,0,0,0,0,1,0,0,0},{0,0,0,1,0,47,99,108,122,135,144,151,159,166,173,177,181,185,188,190,192,193,195,196,198,199,202,202,202,203,204,202,202,201,199,198,197,195,194,193,193,191,190,189,187,184,180,174,166,158,149,139,128,115,98,41,0,0,0,0,0,0,1,0},{0,0,0,0,0,36,91,109,122,133,142,151,158,164,171,176,180,184,188,190,193,196,199,198,200,201,202,203,205,203,204,204,202,202,201,199,199,198,196,194,192,189,188,186,184,180,176,170,163,155,147,137,126,113,97,43,0,0,0,0,0,0,0,0},{0,0,1,0,0,38,93,106,119,131,141,149,156,163,168,173,178,182,186,189,193,196,199,200,201,203,203,204,205,204,203,203,204,203,203,201,201,200,198,197,193,191,188,185,181,177,173,167,159,152,144,135,125,112,97,42,0,0,0,0,0,0,0,0},{0,0,0,0,1,50,98,103,116,129,141,149,155,162,166,172,176,181,185,188,192,196,198,200,201,203,204,204,204,204,204,204,204,204,204,202,201,199,198,197,194,191,187,183,178,174,170,163,156,150,142,133,121,111,96,42,0,0,0,0,0,0,0,0},{0,1,0,1,0,34,86,101,113,126,138,147,154,160,164,170,174,179,183,186,191,195,196,199,201,203,204,204,203,203,202,202,203,203,202,202,201,201,197,195,193,189,185,180,176,173,168,161,155,148,140,129,120,109,95,41,0,0,0,0,0,0,0,0},{0,0,0,0,0,17,74,99,110,123,136,145,153,160,163,169,173,178,181,184,189,193,195,198,199,200,202,202,201,201,200,200,201,202,201,202,201,199,196,194,191,186,182,179,175,171,165,159,153,147,138,128,118,105,93,39,0,0,0,0,0,0,0,0},{0,0,0,0,1,20,72,94,105,120,133,142,150,158,162,167,172,177,180,183,187,190,194,196,198,198,199,199,199,199,199,198,200,200,200,200,200,197,195,192,188,183,179,176,171,167,162,157,150,144,137,125,113,104,90,41,0,0,0,0,0,0,0,0},{0,0,0,1,0,19,71,88,101,118,131,140,148,156,162,167,171,176,178,181,185,188,192,193,196,195,196,195,195,195,195,194,196,196,196,195,195,193,191,188,184,180,177,174,169,165,160,154,148,140,132,121,110,99,82,11,0,0,0,0,0,0,0,0},{0,1,0,0,0,9,63,84,94,113,128,137,146,155,160,165,170,174,176,179,183,186,189,190,193,193,193,193,193,193,193,192,192,192,193,192,190,189,187,185,181,179,175,172,168,164,159,152,145,138,128,118,103,95,77,2,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,48,82,89,109,124,135,145,153,159,165,169,174,177,178,182,184,186,188,189,190,190,190,191,191,191,190,190,189,189,189,188,188,184,181,179,176,172,169,165,161,157,150,143,135,124,112,99,89,39,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,21,69,85,102,119,133,141,150,158,163,169,171,175,178,179,182,183,186,187,187,188,188,189,189,189,189,188,187,187,187,186,185,183,180,178,175,170,166,163,159,154,148,140,131,119,105,92,85,8,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,7,56,80,98,115,126,140,145,156,162,166,170,174,176,178,181,182,183,185,187,187,186,186,187,187,187,188,187,186,185,183,182,181,179,176,172,168,165,161,156,151,144,136,127,112,98,87,66,6,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,1,0,36,75,91,109,125,136,144,150,159,164,169,172,175,177,180,182,182,184,184,185,186,186,186,186,186,185,185,184,184,182,181,179,177,174,170,167,164,160,154,148,141,131,120,107,91,76,19,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,12,67,88,104,119,132,142,149,155,162,167,170,173,176,179,181,182,183,183,183,185,186,185,185,186,183,183,182,182,181,180,178,176,173,169,166,163,158,152,145,137,128,116,100,85,60,5,1,0,0,0,0,0,0,0,0,0},{0,1,0,0,1,0,1,0,42,79,98,114,128,139,146,153,159,164,167,171,174,177,179,180,181,183,183,183,183,183,183,183,182,182,181,180,179,178,176,174,170,168,165,160,155,149,141,132,121,109,90,77,25,0,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,10,70,91,106,123,135,142,149,156,161,164,168,172,175,177,179,179,182,182,182,181,182,182,182,181,180,180,178,178,176,174,172,169,166,161,156,151,144,136,126,114,100,86,46,1,1,0,0,0,0,0,0,0,0,0,0},{0,0,0,1,0,0,0,0,0,28,81,99,116,129,137,146,153,159,162,167,171,174,177,179,179,181,181,182,182,183,182,183,182,181,180,178,178,176,174,172,169,164,158,153,147,140,131,122,110,92,60,12,1,0,0,0,0,0,0,0,0,0,0,0},{2,2,3,2,2,2,2,2,2,6,61,87,103,116,126,134,141,146,151,156,160,162,164,167,168,169,170,171,171,173,172,171,170,170,169,167,166,164,161,159,155,151,146,142,136,127,118,111,97,75,26,2,2,2,3,2,2,2,2,2,2,2,2,2}};

Intensity in a DensityPlot is proportional to the z (depth) from the camera

size=Dimensions[rface][[1]]; hsize=size/2;
ArrayPlot[rface]

Fit continuous 3D function to range surface list

Spline-fit the first surface face to define a continuous function

crface=ListInterpolation[Transpose[rface]];

Plot 3D surface

g2=Plot3D[crface[x,y],{x,1,size},{y,1,size},PlotPoints->32,Lighting->"Neutral",PlotRange->{0,255},Mesh->True,AxesLabel->{"x","y","z"},ViewPoint->{1,-1,3},AspectRatio->1,PlotRange->{{1,size},{1,size},{0,255}},ImageSize->Small]

Calculate surface normals of  surface

Clear[x,y]

nx[x_,y_]:= Evaluate[D[crface[x,y],x]];
ny[x_,y_]:= Evaluate[D[crface[x,y],y]];

nx[x,y] is  ((∂z)/(∂x)), and similarly for ny. The rate of change of depth range is greatest as the face slopes away from the viewpoint, and as we saw for intensity, we can also make a plot of the magnitude of the gradient:

GraphicsRow[{VectorPlot[-{nx[x,y],ny[x,y]},{x,1,size},{y,1,size},VectorPoints->20,VectorScale->.2,ImageSize->Small],DensityPlot[nx[x,y],{x,1,size},{y,1,size}],DensityPlot[Sqrt[nx[x,y]^2+ny[x,y]^2],{x,1,size},{y,1,size}]}]

Lambertian rendering: specification for normals, light, reflectance

Unit surface normals

Normalize the surface normal vectors to unit length:

normface[x_,y_]:=-{nx[x,y],ny[x,y],1}/Sqrt[nx[x,y]^2+ny[x,y]^2+1];

Point light source direction

s is a vector specifying the direction of the light source. The length can be used as to represent intensity. We normalize it.

s={-100,100,100}; s=N[s/Sqrt[s.s]]

Reflectance

The surface is modeled as having constant reflectivity of 1--i.e. "white".

a[x_,y_]:=1.0;

Lambertian rendering model for first surface

imageface[x_,y_]:=a[x,y]*normface[x,y].s;

Render  face surface

DensityPlot[imageface[x,y],{x,1,size},{y,1,size},PlotPoints->32]

Next time

Shape from X, shape from shading

References

Biederman, I. (1985). Human image understanding: Recent research and a theory. Computer Vision, Graphics, and Image Processing, 32, 29-73.
Blake, A., & Bülthoff, H. H. (1990). Does the brain know the physics of specular reflection? Nature, 343, 165-169.
Blake, A., & Bülthoff, H. H. (1991). Shape from specularities: Computation and Psychophysics. Philosophical Transactions of the Royal Society (London) Series B, 331, 237-252.
Blinn, J. F. (1977). Models of light reflection for computer synthesized pictures. Computer Graphics, 11(2), 192-198.
Brady, M., & Yuille, A. (1984). An extremum principle for shape from contour. IEEE Transactions on pattern analysis and machine intelligence, PAMI-6(3), 288-301.
Cook, R., & Torrance, K. (1982). A reflectance model for computer graphics. ACM TOB, 1(1), 7-24.)
Dosher, B. A., Sperling, G., & Wurst, S. A. (1986). Tradeoffs between stereopsis and proximity luminance covariance as determinants of perceived 3D structure. Vision Research, 26(6), 973-990.
Dupuis, P., & Oliensis, J. (1994). An optimal control formulation and related numerical methods for a problem in shape reconstruction. The Annals of Applied Probability, 4(No, 2), 287-346.
Engel, S. A., Remus, D. A., & Sainath, R. (2006). Motion from occlusion. Journal of Vision, 6(5), 9–9. http://doi.org/10.1167/6.5.9
Fantoni, C., & Gerbino, W. (2003). Contour interpolation by vector-field combination. Journal of Vision, 3(4), 281–303. http://doi.org/10.1167/3.4.4
Foley, J. D., van Dam, A., Feiner, S. K., & Hughes, J. F. (1990). Computer Graphics: Principles and Practice (2nd ed.). Reading, Massachusetts. Addison-Wesley.)
Freeman, W. T. (1994). The generic viewpoint assumption in a framework for visual perception. Nature, 368(7 April 1994), 542-545.
W. T. Freeman, Exploiting the generic viewpoint assumption, International Journal Computer Vision, 20 (3), 243-261, 1996. TR93-15. 
W. T. Freeman, The generic viewpoint assumption in a Bayesian framework, in Perception as Bayesian Inference, D. Knill and W. Richards, eds., Cambridge University Press, 365 - 390, 1996.
Greenberg, D. P. (1989). Light reflection models for computer graphics. Science, 244, 166-173.
Greenberg, D. P. (1999 ). A framework for realistic image synthesis. J Commun. ACM, 42(8), 44-53 
Hummel, J. E., & Biederman, I. (1992). Dynamic binding in a neural network for shape recognition. Psychological Review, 99(3), 480–517.Ikeuchi, K., & Horn, B. K. P. (1981). Numerical shape from shading and  occluding boundaries. Artificial Intelligence, 17,, 141-184.
Kersten, D., O'Toole, A. J., Sereno, M. E., Knill, D. C., & Anderson, J. A. (1987). Associative learning of scene parameters from  images. Applied Optics, 26, 4999-5006.)
Kersten, D. (1999). High-level vision as statistical inference. In M. Gazzaniga (Ed.), The Cognitive Neurosciences Cambridge, MA: MIT Press.
Knill, D. C., & Kersten, D. (1990). Learning a near-optimal estimator for surface shape from shading. Computer Vision, Graphics and Image Processing., 50(1), 75-100.
Koenderink, J. J. (1990). Solid Shape . Cambridge, Massachusetts: M.I.T. Press.
Larson, G. W., & Shakespeare, R. (1998). Rendering With Radiance: The Art and Science of Lighting Visualization.
Langer, M. S., & Zucker, S. W. (1994). Shape from Shading on a Cloudy Day. Journal of the Optical Society of America A, 11(2), 467-478.
Lee, J.C.; , "Hacking the Nintendo Wii Remote," Pervasive Computing, IEEE , vol.7, no.3, pp.39-45, July-Sept. 2008, doi: 10.1109/MPRV.2008.53, URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4563908&isnumber=4563896
Lehky, S. R., & Sejnowski, T. J. (1988). Network model of shape from shading: neural function arises from both receptive and projective fields. Nature, 333, 452-454.
Loomis, J. M., Da Silva, J. A., Fujita, N., & Fukusima, S. S. (1992). Visual space perception and visually directed action. J Exp Psychol Hum Percept Perform, 18(4), 906-921.
Mamassian, P., & Kersten, D. (1996). Illumination, shading and the perception of local orientation. Vision Res, 36(15), 2351-2367.
Mamassian, P., Kersten, D., & Knill, D. C. (1996). Categorical local-shape perception. Perception, 25(1), 95-107.
Marotta, J. J., & Goodale, M. A. (2001). Role of familiar size in the control of grasping. J Cogn Neurosci, 13(1), 8-17.
Mingolla, E., & Todd, J. T. (1986). Perception of Solid Shape from Shading. 53, 137-151.
Nakayama, K., & Shimojo, S. (1992). Experiencing and perceiving visual surfaces. Science, 257, 1357-1363.
Oliensis, J. (1991). Uniqueness in Shape From Shading. The International Journal of Computer Vision, 6(2),  75-104.
Ramachandran, V. S. (1988). Perceiving Shape from Shading. Scientific American, , 76-83.
Shirley, Peter, Michael Ashikhmin, Michael Gleicher, Stephen R. Marschner, Erik Reinhard, Kelvin Sung, William B. Thompson, Peter Willemsen (2005) . Fundamentals of Computer Graphics. 2nd edition.
Tarr, M. J., & Bülthoff, H. H. (1995). Is human object recognition better described by  geon-structural-descriptions or by multiple-views? Journal of Experimental Psychology: Human Perception and Performance, 21(6), 1494-1505.
Thompson, William,  Roland Fleming, Sarah Creem-Regehr and Jeanine Kelly Stefanucci (2011). Visual Perception from a Computer Graphics Perspective 
Todd, J. T., & Mingolla, E. (1983). Perception of Surface Curvature and Direction of Illumination from Patterns of Shading. Journal of Experimental Psychology: Human Perception & Performance, 9(4), 583-595.
Todd, J. T., Norman, J. F., Koenderink, J. J., & Kappers, A. M. L. (1997). Effects of texture, illumination, and surface reflectance on stereoscopic shape perception. Perception, 26(7), 807-822.
Todd, J. T., & Reichel, F. D. (1989). Ordinal structure in the visual perception and cognition of smoothly curved surfaces. Psychological Review, 96(4), 643-657.
Yuille, A. L., & Bülthoff, H. H. (1996). Bayesian decision theory and psychophysics. In K. D.C., & R. W. (Ed.), Perception as Bayesian Inference Cambridge, U.K.: Cambridge University Press.
Wilder, J., Feldman, J., & Singh, M. (2011). Superordinate shape classification using natural shape statistics. COGNITION, 119(3), 325–340. doi:10.1016/j.cognition.2011.01.009